Deep learning-based conditional inpainting for restoration of artifact-affected 4D CT images.

BACKGROUND: 4D CT imaging is an essential component of radiotherapy of thoracic and abdominal tumors. 4D CT images are, however, often affected by artifacts that compromise treatment planning quality and image information reliability. PURPOSE: In this work, deep learning (DL)-based conditional inpainting is proposed to restore anatomically correct image information of artifact-affected areas. METHODS: The restoration approach consists of a two-stage process: DL-based detection of common interpolation (INT) and double structure (DS) artifacts, followed by conditional inpainting applied to the artifact areas. In this context, conditional refers to a guidance of the inpainting process by patient-specific image data to ensure anatomically reliable results. The study is based on 65 in-house 4D CT images of lung cancer patients (48 with only slight artifacts, 17 with pronounced artifacts) and two publicly available 4D CT data sets that serve as independent external test sets. RESULTS: Automated artifact detection revealed a ROC-AUC of 0.99 for INT and of 0.97 for DS artifacts (in-house data). The proposed inpainting method decreased the average root mean squared error (RMSE) by 52 % (INT) and 59 % (DS) for the in-house data. For the external test data sets, the RMSE improvement is similar (50 % and 59 %, respectively). Applied to 4D CT data with pronounced artifacts (not part of the training set), 72 % of the detectable artifacts were removed. CONCLUSIONS: The results highlight the potential of DL-based inpainting for restoration of artifact-affected 4D CT data. Compared to recent 4D CT inpainting and restoration approaches, the proposed methodology illustrates the advantages of exploiting patient-specific prior image information.

Self-Supervision for Medical Image Classification: State-of-the-Art Performance with ~100 Labeled Training Samples per Class.

Is self-supervised deep learning (DL) for medical image analysis already a serious alternative to the de facto standard of end-to-end trained supervised DL? We tackle this question for medical image classification, with a particular focus on one of the currently most limiting factor of the field: the (non-)availability of labeled data. Based on three common medical imaging modalities (bone marrow microscopy, gastrointestinal endoscopy, dermoscopy) and publicly available data sets, we analyze the performance of self-supervised DL within the self-distillation with no labels (DINO) framework. After learning an image representation without use of image labels, conventional machine learning classifiers are applied. The classifiers are fit using a systematically varied number of labeled data (1-1000 samples per class). Exploiting the learned image representation, we achieve state-of-the-art classification performance for all three imaging modalities and data sets with only a fraction of between 1% and 10% of the available labeled data and about 100 labeled samples per class.

Clinical application of breathing-adapted 4D CT: image quality comparison to conventional 4D CT.

PURPOSE: 4D CT imaging is an integral part of 4D radiotherapy workflows. However, 4D CT data often contain motion artifacts that mitigate treatment planning. Recently, breathing-adapted 4D CT (i4DCT) was introduced into clinical practice, promising artifact reduction in in-silico and phantom studies. Here, we present an image quality comparison study, pooling clinical patient data from two centers: a new i4DCT and a conventional spiral 4D CT patient cohort. METHODS: The i4DCT cohort comprises 129 and the conventional spiral 4D CT cohort 417 4D CT data sets of lung and liver tumor patients. All data were acquired for treatment planning. The study consists of three parts: illustration of image quality in selected patients of the two cohorts with similar breathing patterns; an image quality expert rater study; and automated analysis of the artifact frequency. RESULTS: Image data of the patients with similar breathing patterns underline artifact reduction by i4DCT compared to conventional spiral 4D CT. Based on a subgroup of 50 patients with irregular breathing patterns, the rater study reveals a fraction of almost artifact-free scans of 89% for i4DCT and only 25% for conventional 4D CT; the quantitative analysis indicated a reduction of artifact frequency by 31% for i4DCT. CONCLUSION: The results demonstrate 4D CT image quality improvement for patients with irregular breathing patterns by breathing-adapted 4D CT in this first corresponding clinical data image quality comparison study.

Deep Learning-Based Automated Thrombolysis in Cerebral Infarction Scoring: A Timely Proof-of-Principle Study.

Background and Purpose: Mechanical thrombectomy is an established procedure for treatment of acute ischemic stroke. Mechanical thrombectomy success is commonly assessed by the Thrombolysis in Cerebral Infarction (TICI) score, assigned by visual inspection of X-ray digital subtraction angiography data. However, expert-based TICI scoring is highly observer-dependent. This represents a major obstacle for mechanical thrombectomy outcome comparison in, for instance, multicentric clinical studies. Focusing on occlusions of the M1 segment of the middle cerebral artery, the present study aimed to develop a deep learning (DL) solution to automated and, therefore, objective TICI scoring, to evaluate the agreement of DL- and expert-based scoring, and to compare corresponding numbers to published scoring variability of clinical experts. Methods: The study comprises 2 independent datasets. For DL system training and initial evaluation, an in-house dataset of 491 digital subtraction angiography series and modified TICI scores of 236 patients with M1 occlusions was collected. To test the model generalization capability, an independent external dataset with 95 digital subtraction angiography series was analyzed. Characteristics of the DL system were modeling TICI scoring as ordinal regression, explicit consideration of the temporal image information, integration of physiological knowledge, and modeling of inherent TICI scoring uncertainties. Results: For the in-house dataset, the DL system yields Cohen's kappa, overall accuracy, and specific agreement values of 0.61, 71%, and 63% to 84%, respectively, compared with the gold standard: the expert rating. Values slightly drop to 0.52/64%/43% to 87% when the model is, without changes, applied to the external dataset. After model updating, they increase to 0.65/74%/60% to 90%. Literature Cohen's kappa values for expert-based TICI scoring agreement are in the order of 0.6. Conclusions: The agreement of DL- and expert-based modified TICI scores in the range of published interobserver variability of clinical experts highlights the potential of the proposed DL solution to automated TICI scoring.

Comparison of intelligent 4D CT sequence scanning and conventional spiral 4D CT: a first comprehensive phantom study.

4D CT imaging is a cornerstone of 4D radiotherapy treatment. Clinical 4D CT data are, however, often affected by severe artifacts. The artifacts are mainly caused by breathing irregularity and retrospective correlation of breathing phase information and acquired projection data, which leads to insufficient projection data coverage to allow for proper reconstruction of 4D CT phase images. The recently introduced 4D CT approach i4DCT (intelligent 4D CT sequence scanning) aims to overcome this problem by breathing signal-driven tube control. The present motion phantom study describes the first in-depth evaluation of i4DCT in a real-world scenario. Twenty-eight 4D CT breathing curves of lung and liver tumor patients with pronounced breathing irregularity were selected to program the motion phantom. For every motion pattern, 4D CT imaging was performed with i4DCT and a conventional spiral 4D CT mode. For qualitative evaluation, the reconstructed 4D CT images were presented to clinical experts, who scored image quality. Further quantitative evaluation was based on established image intensity-based artifact metrics to measure (dis)similarity of neighboring image slices. In addition, beam-on and scan times of the scan modes were analyzed. The expert rating revealed a significantly higher image quality for the i4DCT data. The quantitative evaluation further supported the qualitative: While 20% of the slices of the conventional spiral 4D CT images were found to be artifact-affected, the corresponding fraction was only 4% for i4DCT. The beam-on time (surrogate of imaging dose) did not significantly differ between i4DCT and spiral 4D CT. Overall i4DCT scan times (time between first beam-on and last beam-on event, including scan breaks to compensate for breathing irregularity) were, on average, 53% longer compared to spiral CT. Thus, the results underline that i4DCT significantly improves 4D CT image quality compared to standard spiral CT scanning in the case of breathing irregularity during scanning.

Self-contained deep learning-based boosting of 4D cone-beam CT reconstruction.

PURPOSE: Four-dimensional cone-beam computed tomography (4D CBCT) imaging has been suggested as a solution to account for interfraction motion variability of moving targets like lung and liver during radiotherapy (RT) of moving targets. However, due to severe sparse view sampling artifacts, current 4D CBCT data lack sufficient image quality for accurate motion quantification. In the present paper, we introduce a deep learning-based framework for boosting the image quality of 4D CBCT image data that can be combined with any CBCT reconstruction approach and clinical 4D CBCT workflow. METHODS: Boosting is achieved by learning the relationship between so-called sparse view pseudo-time-average CBCT images obtained by a projection selection scheme introduced to mimic phase image sparse view artifact characteristics and corresponding time-average CBCT images obtained by full view reconstruction. The employed convolutional neural network architecture is the residual dense network (RDN). The underlying hypothesis is that the RDN learns the appearance of the streaking artifacts that is typical for 4D CBCT phase images - and removes them without influencing the anatomical image information. After training the RDN, it can be applied to the 4D CBCT phase images to enhance the image quality without affecting the contained temporal and motion information. Different to existing approaches, no patient-specific prior knowledge about anatomy or motion characteristics is needed, that is, the proposed approach is self-contained. RESULTS: Application of the trained network to reconstructed phase images of an external (SPARE challenge) as well as in-house 4D CBCT patient and motion phantom data set reduces the phase image streak artifacts consistently for all patients and state-of-the-art reconstruction approaches. Using the SPARE data set, we show that the root mean squared error compared to ground truth data provided by the challenge is reduced by approximately 50% while normalized cross correlation of reconstruction and ground truth is improved up to 10%. Compared to direct deep learning-based 4D CBCT to 4D CT mapping, our proposed method performs better because inappropriate prior knowledge about the patient anatomy and physiology is taken into account. Moreover, the image quality enhancement leads to more plausible motion fields estimated by deformable image registration (DIR) in the 4D CBCT image sequences. CONCLUSIONS: The presented framework enables significantly boosting of 4D CBCT image quality as well as improved DIR and motion field consistency. Thus, the proposed method facilitates extraction of motion information from severely artifact-affected images, which is one of the key challenges of integrating 4D CBCT imaging into RT workflows.

4D CT image artifacts affect local control in SBRT of lung and liver metastases.

BACKGROUND AND PURPOSE: 4D CT images often contain artifacts that are suspected to affect treatment planning quality and clinical outcome of lung and liver SBRT. The present study investigates the correlation between the presence of artifacts in SBRT planning 4D CT data and local metastasis control. MATERIALS AND METHODS: The study includes 62 patients with 102 metastases (49 in the lung and 53 in the liver), treated between 2012 and 2016 with SBRT for mainly curative intent. For each patient, 10-phase 4D CT images were acquired and used for ITV definition and treatment planning. Follow-up intervals were 3 weeks after treatment and every 3-6 months thereafter. Based on the number and type of image artifacts, a strict rule-based two-class artifact score was introduced and assigned to the individual 4D CT data sets. Correlation between local control and artifact score (consensus rating based on two independent observers) were analyzed using uni- and multivariable Cox proportional hazards models with random effects. Metastatic site, target volume, metastasis motion, breathing irregularity-related measures, and clinical data (chemotherapy prior to SBRT, target dose, treatment fractionation) were considered as covariates. RESULTS: Local recurrence was observed in 17/102 (17%) metastases. Significant univariable factors for local control were artifact score (severe CT artifacts vs. few CT artifacts; hazard ratio 8.22; 95%-CI 2.04-33.18) and mean patient breathing period (>4.8 s vs. ≤4.8 s; hazard ratio 3.58; 95%-CI 1.18-10.84). Following multivariable analysis, artifact score remained as dominating prognostic factor, although statistically not significant (hazard ratio 10.28; 95%-CI 0.57-184.24). CONCLUSION: The results support the hypothesis that image artifacts in 4D CT treatment planning data negatively influence clinical outcome in SBRT of lung and liver metastases, underlining the need to account for 4D CT artifacts and improve image quality.

Intelligent 4D CT sequence scanning (i4DCT): First scanner prototype implementation and phantom measurements of automated breathing signal-guided 4D CT.

PURPOSE: Four-dimensional (4D) computed tomography (CT) imaging is an essential part of current 4D radiotherapy treatment planning workflows, but clinical 4D CT images are often affected by artifacts. The artifacts are mainly caused by breathing irregularity during data acquisition, which leads to projection data coverage issues for currently available commercial 4D CT protocols. It was proposed to improve projection data coverage by online respiratory signal analysis and signal-guided CT tube control, but related work was always theoretical and presented as pure in silico studies. The present work demonstrates a first CT prototype implementation along with respective phantom measurements for the recently introduced intelligent 4D CT (i4DCT) sequence scanning concept (https://doi.org/10.1002/mp.13632). METHODS: Intelligent 4D CT was implemented on the Siemens SOMATOM go platform. Four-dimensional CT measurements were performed using the CIRS motion phantom. Motion curves were programmed to systematically vary from regular to very irregular, covering typical irregular patterns that are known to result in image artifacts using standard 4D CT imaging protocols. Corresponding measurements were performed using i4DCT and routine spiral 4D CT with similar imaging parameters (e.g., mAs setting and gantry rotation time, retrospective ten-phase reconstruction) to allow for a direct comparison of the image data. RESULTS: Following technological implementation of i4DCT on the clinical CT scanner platform, 4D CT motion artifacts were significantly reduced for all investigated levels of breathing irregularity when compared to routine spiral 4D CT scanning. CONCLUSIONS: The present study confirms feasibility of fully automated respiratory signal-guided 4D CT scanning by means of a first implementation of i4DCT on a CT scanner. The measurements thereby support the conclusions of respective in silico studies and demonstrate that respiratory signal-guided 4D CT (here: i4DCT) is ready for integration into clinical CT scanners.

Skin Lesion Classification Using CNNs With Patch-Based Attention and Diagnosis-Guided Loss Weighting.

OBJECTIVE: This paper addresses two key problems of skin lesion classification. The first problem is the effective use of high-resolution images with pretrained standard architectures for image classification. The second problem is the high-class imbalance encountered in real-world multi-class datasets. METHODS: To use high-resolution images, we propose a novel patch-based attention architecture that provides global context between small, high-resolution patches. We modify three pretrained architectures and study the performance of patch-based attention. To counter class imbalance problems, we compare oversampling, balanced batch sampling, and class-specific loss weighting. Additionally, we propose a novel diagnosis-guided loss weighting method that takes the method used for ground-truth annotation into account. RESULTS: Our patch-based attention mechanism outperforms previous methods and improves the mean sensitivity by [Formula: see text]. Class balancing significantly improves the mean sensitivity and we show that our diagnosis-guided loss weighting method improves the mean sensitivity by [Formula: see text] over normal loss balancing. CONCLUSION: The novel patch-based attention mechanism can be integrated into pretrained architectures and provides global context between local patches while outperforming other patch-based methods. Hence, pretrained architectures can be readily used with high-resolution images without downsampling. The new diagnosis-guided loss weighting method outperforms other methods and allows for effective training when facing class imbalance. SIGNIFICANCE: The proposed methods improve automatic skin lesion classification. They can be extended to other clinical applications where high-resolution image data and class imbalance are relevant.

Intelligent 4D CT sequence scanning (i4DCT): Concept and performance evaluation.

PURPOSE: Four-dimensional (4D) CT imaging is a central part of current treatment planning workflows in 4D radiotherapy (RT). However, clinical 4D CT image data often suffer from severe artifacts caused by insufficient projection data coverage due to the inability of current commercial 4D CT imaging protocols to adapt to breathing irregularity. We propose an intelligent sequence mode 4D CT imaging protocol (i4DCT) that builds on online breathing curve analysis and respiratory signal-guided selection of beam on/off periods during scan time in order to fulfill projection data coverage requirements. i4DCT performance is evaluated and compared to standard clinical sequence mode 4D CT (seq4DCT) and spiral 4D CT (spiral4DCT) approaches. METHODS: i4DCT consists of three main blocks: (a) an initial learning period to establish a patient-specific reference breathing cycle representation for data-driven i4DCT parameter selection, (b) online respiratory signal-guided sequence mode scanning (i4DCT core), (c) rapid breathing record analysis and quality control after scanning to trigger potential local rescanning (i4DCT rescan). Based on a phase space representation of the patient's breathing signal, i4DCT core implements real-time analysis of the signal to appropriately switch on and off projection data acquisition even during irregular breathing. Performance evaluation was based on 189 clinical breathing records acquired during spiral 4D CT scanning for RT planning (data acquisition period: 2013-2017; Siemens Somatom with Varian RPM system). For each breathing record, i4DCT, seq4DCT, and spiral4DCT scanning protocol variants were simulated. Evaluation measures were local projection data coverage ß cov ; number ϵ total of local projection data coverage failures; and number ϵ pat of patients with coverage failures; average beam on time t beam on as a surrogate for imaging dose and total patient on table time t table as the time between first and last beam on signal. RESULTS: Using i4DCT, mean inhalation and exhalation projection data coverage ß cov increased significantly compared to standard spiral 4D CT scanning as applied for the original clinical data acquisition and conventional 4D CT sequence scanning modes. The improved projection data coverage translated into a reduction of coverage failures ϵ total by 89% without and 93% when allowing for a rescanning at up to five z-positions compared to spiral scanning and between 76% and 82% without and 85% and 89% with rescanning when compared to seq4DCT. Similar numbers were observed for ϵ pat . Simultaneously, i4DCT (without rescanning) reduced the beam on time on average by 3%-17% compared to standard spiral 4D CT. In turn, the patient on table time increased by between 35% and 66%. Allowing for rescanning led on average to additional 5.9 s beam on and 10.6 s patient on table time. CONCLUSIONS: i4DCT outperformed currently implemented clinical fixed beam on period 4D CT scanning approaches by means of a significantly smaller data coverage failure rate without requiring additional beam on time compared to, for example, conventional spiral 4D CT protocols.